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When  Lt  Gen  Charlie  Groom  took  over  as  the  Director  of  the  Defense  Information  Systems 
Agency  (DISA)  in  July  2005,  he  brought  us  a  new  message:  “It’s  All  about  Speed.”  What  he 
meant  was  simply  this:  It  takes  us  too  long  to  get  new  capabilities  into  the  hands  of  the 
warfighters.  When  he  retired  this  past  summer,  3  years  after  his  arrival,  he  left  a  legacy  of 
change — of  innovation — in  how  we  acquire  and  test  information  technologies  (IT)  in  DISA. 
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eneral  Croom  was  right.  My  experi¬ 
ence  with  acquisition  and  testing  of 
information  technologies  began  in 
1998  on  my  arrival  in 
the  Army  Test  and 
Evaluation  Command  and  my  assign¬ 
ment  as  Evaluator  for  one  of  the  Army’s 
“digitization”  systems.  Six  days  after  my 
arrival,  I  found  myself  at  Et.  Hood, 

Texas,  seeing  the  new  capability  for  the 
first  time.  I  was  amazed  at  this  new 
system  and  wished  I’d  had  it  in  my  units 
way  hack  when.  Six  years  later,  we  hadn’t 
managed  to  get  that  system  through  the 
acquisition  process,  had  not  even  com¬ 
pleted  the  Initial  Operational  Test  and 
Evaluation  (IOT8cE).  It  took  the  oper¬ 
ational  necessity  of  the  second  Gulf  War 
to  get  that  system  into  units  other  than 
the  test  unit.  And  when  we  did  that,  we  had  to  spread 
the  few  systems  we  had  for  testing  out  to  those  other 
units  and  didn’t  give  them  the  luxury  of  time  to  do  a  lot 
of  training.  But  what  a  remarkable  difference  that  Blue 
Force  Tracking  system  made  for  the  warfighters. 

There  are  many  reasons  why  we  took  more  than  6  years 
to  field  the  system;  hindsight  suggests  to  me  that  none  of 
them  was  particularly  good.  There  were  other  ways  to 
develop,  test,  and  field  the  new  system;  we  just  didn’t  look 
“outside  the  box”  of  DoD  5000  to  find  them.  That’s  the 
message  General  Croom  brought  to  Defense  Information 
Systems  Agency  (DISA).  When  he  looked  at  how 
industry,  especially  companies  like  Google,  eBay,  Ama¬ 
zon,  and  Travelocity,  just  to  name  a  few,  brought  new 
capabilities  to  its  customers  in  short  cycles — ^with  speed — 
he  asked  why  the  Department  of  Defense  (DoD)  couldn’t 
do  the  same.  The  fact  is,  we  can,  but  it  requires  some 


fundamental  changes  in  acquisition  philosophy.  It  is  time 
that  we  took  a  hard  look  at  how  we  acquire  and  test 
information  technologies  in  the  DoD. 

Adopt — buy — create 

Shortly  after  his  arrival,  Lt  Gen 
Croom  cast  his  message  of  speed  in  these 
terms:  “Adopt  before  Buy,  Buy  before 
Create” — the  ABCs.  It  was  a  simple 
message;  to  speed  up  the  process  of 
getting  enhanced  capabilities  into  the 
hands  of  the  soldiers,  sailors,  airmen,  and 
marines  that  need  them,  we  would  look 
first  for  something  already  available  in 
the  Department — say  an  Army  system 
for  example,  that  would  satisfy  a  need 
identified  by  the  Navy — and  adopt  it  for 
fielding  to  the  entire  Department.  If 
there  was  no  capability  already  fielded, 
then  we  would  look  for  a  commercial  product — maybe 
even  “the  80%  solution” — and  buy  it  for  fielding  to  the 
Enterprise,  then  add  capability  in  short  cycles.  As  a  last 
recourse,  we  would  create  it;  last  because  that  approach 
comes  with  a  lot  of  program  management  overhead, 
cumbersome  decision-making  processes,  and  some¬ 
times  heavy-handed  oversight,  not  to  mention  lengthy 
periods  of  development  and  testing — processes  that  are 
slow  to  move  and  adjust  when  it’s  all  about  speed. 

There  were  other  innovations  in  this  new  acquisition 
paradigm.  One  notable  innovation  was  to  bring 
competition  into  the  acquisition  process;  the  theory 
being  that  if  there  is  more  than  one  provider  of  a 
capability,  and  those  providers  make  money  based  on 
product  use  within  DoD,  then  competition  for  market 
share  will  motivate  those  providers  to  continually 
improve  their  products  and  entice  more  users  to  their 
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Table  1.  Test  and  evaluation  for  the  ABCs 

IT  acquisition  strategy 

Capability  maturity/risk 

Critical  T&E  issues 

Adopt 

Capability  in  use  in  Department  of  Defense 

Scalable  performance  and  support 

Buy 

Capability  in  use  in  commercial  sector 

Scalable  performance  and  support 

Secure 

Interoperable 

Create 

New  capability  to  be  developed 

Scalable  performance  and  support 

Secure 

Interoperable 

Effective,  suitable,  survivable 

side.  It’s  an  interesting  idea  that  DISA  has  put  to  the 
test,  and  we  are  starting  to  see  it  work  in  the  Net 
Centric  Enterprise  Services  (NCES)  program. 

Those  of  us  in  the  test  and  evaluation  (T6cE) 
business  need  to  hear,  loud  and  clear,  the  message  of 
speed  because  we  can  ill-afford  to  be  an  obstacle  in  the 
path  of  bringing  capability  improvements  to  the 
warfighters.  Instead,  we  need  to  be  an  enabler  in  the 
process  that  ensures  rapid  delivery  of  effective,  suitable, 
interoperable,  and  secure  information  technologies. 
That  type  of  agility  can  only  occur  when  we  are 
involved  from  the  beginning.  In  some  commercial 
circles,  they  refer  to  this  as  “test  driven  development.” 

T&E  for  the  ABCs 

The  ABCs  present  an  opportunity  for  innovation 
and  invention  in  T&E.  Once  we  have  identified  a 
capability  need,  through  what  is  now  the  Joint 
Capabilities  Integration  and  Development  System 
(JCIDS),  the  program  manager  formulates  an  acqui¬ 
sition  strategy.  Eor  an  IT  system,  the  acquisition 
strategy  is  essentially  a  choice  among  the  ABCs — 
adopt  what’s  available  already,  buy  it,  or  create  it. 
Likewise,  we  should  have  a  T&E  strategy  that 
corresponds  to  the  ABCs. 

If  the  acquisition  approach  is  to  adopt  something 
already  available  in  the  DoD,  then  that  capability  has 
presumably  negotiated  aU  of  the  acquisition  and  T&E 
wickets  to  achieve  its  fielding  decision.  More  specific  to 
T&E,  that  capability  has  already  been  determined  to  be 
effective  and  suitable  for  its  intended  use.  As  a  capability 
proposed  for  the  enterprise,  however,  there  are  two 
relevant  issues  to  resolve  before  full  deployment: 

•  scalability  of  performance  (Does  the  capability 
still  perform  at  acceptable  levels  under  greater  use 
at  the  enterprise  level?), 

•  scalability  of  support  (Is  there  sufficient  capacity 
for  supporting  the  capability  at  the  enterprise 
level,  such  as  help  desk  capacity?). 

There  may  be  other  considerations,  but  the  motiva¬ 
tion  behind  the  adopt  strategy  is  to  accept  the  risk  and 


make  an  existing  capability  available  to  a  broader  user 
base. 

In  the  case  of  the  “buy”  approach,  the  premise  is  that 
we  have  identified  a  commercial  product  that  satisfies 
all  or  part  of  the  need  identified  in  JCIDS.  The 
product  is  already  in  the  commercial  marketplace,  but 
more  specifically  to  T&E,  it  has  satisfied  unit  and 
functional  testing  by  the  vendor.  If  we  accept  the 
capabilities  and  limitations  of  the  commercial  product 
as  is,  then  the  remaining  issues  for  us  to  verify  prior  to 
use  in  the  DoD  environment  are: 

•  performance  and  support  at  the  enterprise  level, 

•  interoperability  with  other  DoD  systems  or 
services, 

•  security  (information  assurance). 

Focusing  T&E  resources  on  these  areas  will  permit 
rapid  assessment  and  recommendations  for  the  acqui¬ 
sition  decision  makers. 

In  the  case  of  the  “create”  approach,  no  existing 
capability  in  the  Department  or  commercial  sector 
satisfies  enough  of  the  identified  need.  In  this  case,  the 
capability  must  be  developed,  and  T&E  wiU  have  to 
answer  all  standard  evaluation  concerns.  Table  1 
summarizes  the  T&E  concept  for  the  ABCs. 

However,  the  create  approach  must  not  be  “business 
as  usual”  for  DoD  acquisitions.  The  key  for  IT 
acquisition  is  to  bring  new  capabilities  forward  in 
small,  warfighter-relevant  increments,  or  “sprints.”  In 
the  commercial  sector,  some  refer  to  this  process  as 
“Agile  development.”  There  is  a  wealth  of  information 
available  about  agile  processes,  so  I  will  not  attempt  to 
describe  it  in  detail  here.  At  the  core  of  this  process, 
however,  is  the  idea  that  a  small  team  of  developers, 
users,  and  testers  work  together  to  define,  build,  test, 
and  field  new  capabilities  in  short  cycles — “build  a 
little,  test  a  little,  field  a  little”  as  General  Croom 
would  say.  To  field  the  system,  we  would  start  small 
and  scale  rapidly,  with  T&E  monitoring  to  ensure 
capability  effectiveness  as  use  scales  upward. 

There  are  some  fundamental  differences  in  the  ABC 
approaches  when  compared  to  current  acquisition 
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Table  2.  Test  and  evaluation  in  the  Department  of  Defense  acquisition  process 

Activity 

Test  agent 

Conditions 

Customer 

Reference 

Developmental  T&E 

PMO/ contractor/ 
government  DT 
organization 

As  determined  by  PMO; 
generally  benign,  lab; 
developer  personnel 

PMO 

DOD  5000 

Operational  T&E 

OTA 

“Operationally  realistic,..., 
typical  users” 

MDA 

Title  10 

DoD  5000 

Joint  Interoperability  Test 
Certification 

JITC 

“Applicable  capability 
environments” 

J6 

DODD  4630.5 

DODI  4630.08 

CJCSI  6212.01D 

Security  T&E  (lA 
Certification  & 
Accreditation) 

OTA,  DIA,  FSO,  NSA 

Operational,  lab 

DAA 

DoDI  8510.01 

DIACAP* 

PMO,  Program  Management  Office;  DT,  Developmental  Test;  OTA,  Operational  Test  Agency;  MDA,  Milestone  Decision  Authority;  JITC, 
Joint  Interoperability  Test  Command;  J6,  Joint  Staff  J6  is  Director  for  Command,  Control,  Communications,  and  Computer  Systems;  DoDD, 
DoD  Directive;  DoDI,  DoD  Instruction;  CJCSI,  Chairman  Joint  Chiefs  of  Staff  Instruction;  lA,  Information  Assurance;  DIA,  Defense 
Intelligence  Agency;  FSO,  Field  Security  Office  (DISA);  NSA,  National  Security  Agency;  DAA,  Designated  Approving  Authority;  DIACAP, 
Defense  Information  Assurance  Certification  and  Accreditation  Process;  DOT6cE,  Director,  Operational  Test  and  Evaluation. 

*  Note  also  the  DOT&E  Policy  on  testing  lA  during  OT&E.  DIACAP  C&A  does  not  complete  the  requirement  for  lA  testing. 


practice.  The  ABC  model  accepts  risk,  whereas  our 
traditional  model  is  founded  on  risk  aversion.  The 
current  scheme  of  acquisition  milestones  are  not  a  good 
fit  in  the  ABC  model — the  ABC  acquisition  process  is 
too  fast.  Our  traditional  acquisition  decision-making 
processes  may  need  to  change;  for  example,  in  this 
model,  there  would  he  no  full  deployment  decision 
review.  Likewise,  our  T&E  practices  should  adjust.  For 
example,  in  none  of  the  T&E  approaches  suggested  is 
there  a  concept  of  a  large-scale  lOT&E  or  Capstone 
event.  For  IT  systems,  the  lOT&E  as  we  think  of  it 
today  is  an  obsolete  practice. 

None  of  this  suggests  we  eliminate  oversight  or 
testing.  Each  has  a  critical  role,  hut  we  should 
acknowledge  that  the  processes  we’ve  built  and  put  in 
place  for  the  past  decades,  and  which  have  always  been 
focused  on  major  defense  systems  such  as  tanks,  ships, 
and  planes,  may  not  be  well  suited  for  the  agile  IT 
environment.  We  should  look  to  the  commercial  IT 
sector  and  pull  their  good  ideas  into  the  DoD.  And  we 
should  teach  innovative  IT  acquisition  concepts,  such  as 
agile  development  and  test,  to  our  program  managers 
and  testers  as  part  of  our  formal  acquisition  curriculum. 

T&E  for  better  decision  making 

There  are  at  least  four  different  test  and  evaluation 
activities  that  support  different  decision-making  pro¬ 
cesses  for  information  technologies,  but  the  question  is, 
do  the  four  activities  improve  our  acquisition  decision¬ 
making  process?  The  T&E  activities  include 

•  Developmental  Test  and  Evaluation  (DT&E) 

•  Operational  Test  and  Evaluation  (OT&E) 

•  Joint  Interoperability  Test  and  Certification 


•  Security  Test  and  Evaluation  (Information  As¬ 
surance  Certification  and  Accreditation) 

We  do  each  of  these  tests  for  different  purposes,  and 
that  is  certainly  understandable.  What  is  not  under¬ 
standable  is  why  these  tests  are  performed  under 
different  conditions,  by  different  test  agents,  for 
different  customers.  Developmental  testing,  for  exam¬ 
ple,  helps  the  program  manager  find  and  fix  problems, 
ensure  compliance,  and  improve  production  processes. 
It  tends  to  be  more  technical  than  operational.  Robust 
DT  helps  ensure  readiness  for  OT.  OT,  on  the  other 
hand,  ensures  readiness  for  fielding.  Why  doesn’t  DT 
ensure  readiness  for  fielding? 

Interoperability  and  security  testing  are  more 
specialized  and  feed  other  decision-making  processes, 
i.e..  Joint  Interoperability  Certification  and  the  De¬ 
fense  Information  Assurance  Certification  and  Ac¬ 
creditation  Process.  Unfortunately,  we  do  not  treat 
these  two  processes  as  integral  to  acquisition  decision 
making,  which  results  in  situations  in  which  the 
Milestone  Decision  Authority  might  approve  a  deci¬ 
sion  to  buy  for  the  Department,  while  the  Designated 
Approving  Authority  (DAA)  may  not  authorize  its 
operation  on  their  network.  Table  2  summarizes  the 
T&E  landscape  for  IT. 

Our  acquisition  decision  making  would  be  much 
improved  if  the  various  T&E  activities  fit  into  a 
holistic  model.  Recent  emphasis  on  “integrated  T&E,” 
such  as  written  in  the  December  22,  2007,  memoran¬ 
dum  signed  by  the  Director  of  Test  and  Evaluation  and 
the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics,  acknowledges  the  impor¬ 
tance  of  early  involvement  of  the  test  community,  but 
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does  not  do  enough  to  eliminate  the  barriers  that  exist 
between  test  activities  or  compel  streamlined  T6cE. 

Capability  test  and  evaluation 

In  DISA,  we  are  working  to  unite  all  test  activities 
into  a  holistic,  coherent  T&E  model.  We  refer  to  this 
model  as  capability  T&E  (CT&E).  For  capabilities 
being  developed  in  sprints,  having  four  different 
organizations  doing  the  testing  at  different  times, 
under  different  conditions,  and  writing  different 
reports  is  laughably  inefficient.  The  commercial  sector 
would  never  do  this. 

To  the  extent  possible,  we  would  like  to  designate  a 
capability  test  team  (CTT)  to  plan  and  conduct  CT&E 
events.  All  CT&E  events  are  a  shared  resource.  To 
ensure  agility  in  T&E,  CT&E  events  are  risk-based, 
according  to  which  ABC  acquisition  approach  is  used. 
For  each  sprint  there  is  one  CT&E  event.  CT&E  can 
be  thought  of  as  a  “one  team,  one  time,  one  set  of 
conditions”  approach  to  T&E.  Upon  completion,  the 
CTT  writes  one  report  for  use  by  all  decision  makers: 
the  milestone  decision  authority,  the  interoperability 
certifier,  and  the  D7\A.  One  means  to  obtain  buy-in 
for  this  concept  would  be  to  have  all  of  these  decision 
makers  sign  the  T&E  master  plan  (TEMP). 

To  ensure  acceptance  by  all  decision  makers,  CT&E 
test  designs  must  also  be  mission-focused.  During  the 
CT&E,  typical  users  exercise  the  capability  under  test, 
similar  to  beta  testing  in  the  commercial  sector,  and  are 
supported  as  intended  when  fielded.  The  combat 
developer,  part  of  the  CTT,  defines  and  validates  the 
scenario  and  mission  threads.  The  test  conditions 
replicate  the  operating  environment,  leveraging  dis¬ 
tributed  live,  virtual,  and  constructive  capabilities,  such 
as  the  Joint  Mission  Environment  Test  Capability 
(JMETC),  to  the  maximum  extent  possible.  CT&E 
therefore  expands  on  and  puts  into  practice  the  concept 
of  “integrated  T&E”  by  including  all  stakeholders  in 
developing  the  test  strategy  at  the  beginning  of  the 


acquisition  process  and  by  structuring  all  test  activities 
as  shared  resources. 

Some  organizations  will  see  CT&E  as  an  infringe¬ 
ment  on  their  independence.  There  is  nothing  about 
the  CT&E  concept  that  precludes  CTT  members  from 
performing  independent  evaluation.  In  fact,  the 
CT&E  construct  works  best  when  all  stakeholders 
have  their  say.  The  TEMP  should  reflect  the  ABC 
strategy  being  followed  and  describe  the  CT&E  events 
designed  to  ensure  that  critical  issues  are  adequately 
addressed.  Once  approved,  the  CTT  executes. 

We  can  fundamentally  change  the  way  we  acquire 
and  test  information  technologies  in  the  Department. 
By  focusing  on  small  improvements  to  capability, 
development  cycles  in  sprints,  and  a  one  team,  one 
time,  one  set  of  conditions  T&E  model,  we  can 
simultaneously  reduce  time  to  fielding  while  improving 
product  quality.  After  all,  it’s  all  about  speed.  □ 


Dr.  Steven  J.  Hutchison,  a  member  of  the  Senior 
Executive  Service,  is  the  Test  and  Evaluation  Executive  for 
the  Defense  Information  Systems  Agency.  He  is  responsible 
for  developing  and  enforcing  T^E  policy  and  procedures, 
representing  DISA  to  the  DoD  T^E  community,  and 
providing  direct  support  to  DISA  programs  for  T^E  of  IT 
capabilities.  Prior  to  his  arrival  at  DISA,  Dr.  Hutchison 
served  in  the  office  of  the  Director,  Operational  Test  and 
Evaluation  (DOT^E),  Office  of  the  Secretary  of  Defense,  as 
a  net-centric  warfare  systems  analyst.  While  in  DOT^AE,  he 
had  oversight  responsibilities  for  several  of  the  major 
wafighting  information  systems,  including  the  Global 
Command  and  Control  System — -Joint,  the  Service  variants 
of  the  Distributed  Common  Ground/Surface  System,  and  the 
Net  Enabled  Command  Capability.  Dr  Hutchison  also 
served  for  6  years  in  the  Army  Test  and  Evaluation 
Command  as  an  evaluation  officer  and  later  as  the  assistant 
technical  director.  E-mail:  steven.hutchison@disa.mil 


10  /7EA  Journal 


